Visual task feedback for workstations in materials handling facilities

ABSTRACT

Visual task feedback for workstations in a materials handling facility may be implemented. Image data of a workstation surface may be obtained from image sensors. The image data may be evaluated with regard to the performance of an item-handling task at the workstation. The evaluation of the image data may identify items located on the workstation surface, determine a current state of the item-handling task, or recognize an agent gesture at the workstation. Based, at least in part on the evaluation, one or more visual task cues may be selected to project onto the workstation surface. The projection of the selected visual task cues onto the workstation surface may then be directed.

BACKGROUND

This application is a continuation of U.S. patent application Ser. No.15/083,228, filed Mar. 28, 2016, now U.S. Pat. No. 9,552,635, which is acontinuation of U.S. patent application Ser. No. 14/228,127, filed Mar.27, 2014, now U.S. Pat. No. 9,299,013, which are hereby incorporated byreference herein in their entirety.

The efficient movement, storage, and control of items is more importantthan ever. Advances in communicating orders for items have led toincreased expectations as to the promptness in which these orders arefulfilled. Various suppliers, merchants, distributors, or otherconveyors of goods, may operate materials handling facilities to storeand move items to satisfy received orders as part of a fulfillmentnetwork. For example, electronic marketplaces, such as those accessiblevia the Internet, may include a catalog of items or products availablefor purchase. These items may be offered as the basis for commerce(e.g., sale or trade). Customers may utilize a web browser to visit amerchant's website, select an item for purchase from the catalog, andengage in a checkout process to finalize an order for the item. Themerchant may operate a fulfillment network including various materialshandling facilities in order to process such orders, such as a facilitythat maintains an inventory of items, picks the ordered item frominventory, and prepares shipments of the purchased item. A shipmentcarrier may acquire such shipments from the merchant and deliver theshipments to the respective purchasing customers.

Improvements to materials handling systems have increased the throughputin which items may be processed at materials handling facilities. Forinstance, automation techniques for processes such as storing andretrieving items from inventory, as well as conveying items within amaterials handling facility, may improve the performance of theseprocesses. Some processes, however, are less susceptible to improvementslike automation. Consider processes with manually performed or assistedtasks; performance of the process may be limited to the capabilities ofa human agent performing the respective task. As the capabilities ofdifferent human agents may vary widely, processes with manuallyperformed or assisted tasks can be subject to inconsistent performance.When combined with processes operating with improved performancecharacteristics, like the aforementioned automation techniques,processes with manually performed or assisted tasks may reduce theoverall effectiveness of such techniques.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is series of block diagrams illustrating visual task feedback forworkstations in a materials handling facility, according to someembodiments.

FIG. 2 is a block diagram illustrating a logical representation of amaterials handling facility, according to some embodiments.

FIG. 3 is a block diagram illustrating a logical illustration of aworkstation visual feedback system, according to some embodiments.

FIG. 4 is a diagram illustrating a workstation implementing aworkstation visual feedback system, according to some embodiments.

FIGS. 5A-5F are diagrams illustrating different visual cues projectedonto a pack workstation surface, according to some embodiments.

FIG. 6 is a high-level flowchart illustrating various methods andtechniques for performing visual feedback for workstations in materialshandling facilities, according to some embodiments.

FIG. 7 is a high-level flowchart illustrating various methods andtechniques for performing item recognition for items located on aworkstation surface, according to some embodiments.

FIG. 8 is a high-level flowchart illustrating various methods andtechniques for selecting a visual task cue to be projected onto aworkstation surface, according to some embodiments.

FIG. 9 is a high-level flowchart illustrating various methods andtechniques for adjusting projected visual task cues based on additionalimage and geometric information for a workstation surface, according tosome embodiments.

FIG. 10 is a high-level flowchart illustrating various methods andtechniques for projecting a visual task cue based on a viewingperspective of an agent at a workstation, according to some embodiments.

FIG. 11 is a block diagram illustrating an example computing system,according to some embodiments.

While embodiments are described herein by way of example for severalembodiments and illustrative drawings, those skilled in the art willrecognize that the embodiments are not limited to the embodiments ordrawings described. It should be understood, that the drawings anddetailed description thereto are not intended to limit embodiments tothe particular form disclosed, but on the contrary, the intention is tocover all modifications, equivalents and alternatives falling within thespirit and scope as defined by the appended claims. The headings usedherein are for organizational purposes only and are not meant to be usedto limit the scope of the description or the claims. As used throughoutthis application, the word “may” is used in a permissive sense (i.e.,meaning having the potential to), rather than the mandatory sense (i.e.,meaning must). Similarly, the words “include”, “including”, and“includes” mean including, but not limited to.

DETAILED DESCRIPTION

The systems and methods described herein may implement visual taskfeedback for workstations in a materials handling facility, according tovarious embodiments. Materials handling facilities may implementmultiple processes to provide for the movement, storage, and/or controlof items within the materials handling facilities. Some of theseprocesses may include manually performed or directed item-handling tasksby operators, agents, or associates. In order to complete item-handlingtasks, agents may receive information from many different sources. Forinstance, monitors, or other displays may show text, such asinstructions. Other components or objects, such as physical documents,worksheets, tags, or manuals, as well as scattered indicators may conveysome information pertinent to the performance of a task. As the processevolves to increase efficiency, agents may have less time to view orread information from multiple sources.

In various embodiments, a visual feedback system for a workstation maybe implemented, streamlining and/or reducing the number of informationsources an agent views in order to complete a task. For example, insteadof dividing an agent's attention when performing a scanning procedure ofitems by scanning an item and then looking at a separate display toconfirm the scan completed successfully, a visual cue indicating successor failure of a particular operation may be projected within the view ofthe agent such that the agent need not look away from the particularitem or area being viewed as part of performing the operation. Differentvisual cues may indicate or provide the information necessary to performa respective item-handling task so that viewing multiple differentinformation sources may no longer be necessary for completion.

Item(s) upon which item-handling tasks are performed may be located on aworkstation surface, such as a table, desk, shelf, or other surface, invarious embodiments. A visual feedback system for the workstationsurface may provide an augmented reality interface for the workstation,such that tasks performed at the workstation surface may be augmentedaccording to the visual feedback system. In various embodiments,different imaging techniques may be implemented to capture image datafor a workstation surface (e.g., a multi-model imaging RGBZ sensor).Based, at least in part, on the captured imaging data, individualproduct items on the workstation surface may be detected, segmented, andrecognized using techniques including, but not limited to, imagerecognition/matching and bar-code recognition. Based, at least in part,on the identified items, visual cues indicating information pertainingto the items and/or item-handling task may be determined and projected.Image projection systems may then project the determined visual cuesonto the workstation surface, to provide information to the agent.

FIG. 1 is series of block diagrams illustrating visual task feedback forworkstations in a materials handling facility, according to someembodiments. Workstation 120 may be a workstation for the performanceone or more item-handling tasks in a materials handling facility(various examples of which are given below with regard to FIG. 2, suchas picking, stowing, packing, or sorting). Different items 150, such asitems 150 a, 150 b, and 150 c, may be collected and/or located on aworkstation surface 122 which may be a desk, table, shelf, or any othersurface for holding items 150 as part of performing the item-handlingtask. Agent 140 may be responsible for performing or directing theperformance of the item-handling task with regard to items 150.

Workstation visual feedback system 110 may be implemented at workstation120 in order to perform various techniques for providing visual feedbackfor item-handling tasks performed by agent 140. In some embodiments,workstation visual feedback system may include one or more imagingand/or depth sensors, scales, projectors, audio speakers, agent trackingsensors, or other components to implement visual feedback techniques.Components of workstation visual feedback system 110 may be fixed, asillustrated in FIG. 1, at a particular station, such as mounted orlocated above workstation surface 122. While in other embodiments, somecomponents of workstation visual feedback system 110 may be separatelylocated, with projection components, for instance located on theworkstation surface or, using wearable technology, located on or worn byagent 140.

As illustrated in scene 102, imaging sensors (not shown) may capturesurface image and/or corresponding geometric (e.g. as may be determinedbased on depth data) data 112 of workstation surface 122. For example, ahigh-resolution camera implemented as part of workstation visualfeedback system 110 may capture a real-time or near real-time stream ofimage data of a workstation surface. Items 150, or other objects withinthe field of view for imaging sensors, may be captured. For instance, asillustrated in scene 102, items 150 a, 150 b, and 150 c are capturedwithin the field of view of the image sensor(s). In some embodiments,geometric information, either captured through a separate or the sameimaging sensor, corresponding to the image data that may be obtained.For example, a time-of-flight camera may provide depth datacorresponding to the distance between the camera and the object in eachportion of the image which may be used to construct geometry informationfor the workstation surface. Thus, geometric information correspondingto the image data of workstation surface 122 may be able to distinguishbetween the relative heights of item 150 a, surface 122, item 150 b, anditem 150 c, each of which may be different.

Once image data for workstation surface 122 is captured, the image datamay be evaluated to identify items 150 located on workstation surface122. For example, item matching techniques may be employed by accessinga data store that maintains item description information, such as itemimage information, upon which a comparative analysis may be performed toidentify the item. In some embodiments, additional information, such asgeometric information or weight information of items 150 may also beused to identify items 150. In some embodiments, the image data may beevaluated to identify a current state of the item-handling task beingperformed at workstation 120, which may provide the state of, or otherinformation concerning the progress of an item-task. For example, at apacking workstation, a packing container (e.g., a box) may be identifiedbased on the captured image data, indicating that a packing step is tobegin, in progress, or complete. Evaluation of the image data may alsobe performed, in various embodiments to recognize agent gestures withregard to the performance of the item-handling task. For instance,certain motions may be recognized as various steps in the performance ofan assembly task or detected as input for user interfaces (which may beprojected onto workstation surface 122).

Once the image data is evaluated, various different visual task cues maybe selected or determined to project onto workstation surface 122, insome embodiments, based, at least in part on the evaluation of the imagedata. For example, scene 104 illustrates a visual cue projection 114onto workstation surface 122 (and/or onto item 150 a) to illustrate avisual cue 116 with regard to item 150 a. Visual cue 116 may conveyinformation to agent 140 about item 150 a, such as whether the item isincorrect (e.g., as illustrated in scene 106), extraneous,unrecognizable (which may require further recognition action, such asmanually scanning or identifying the item), or particular taskinstructions for the item. In various embodiments, projection of visualcue 114 may be performed by one or more image projection systems todisplay visual cue 116 according to the image data of the workstationsurface. For instance, the projection may be registered with theoriginal scene of image data upon which the selection and projection ofthe visual cue was based. The visual cue may be projected toward thelocation of item 150 a, for example, based on the location of item 150 aon workstation surface 122 as described in the captured image data 112.In some embodiments, multiple projection systems may be used to projectvisual cue 114 in order to adjust the projection 114 in the event of achange in workstation surface 122 (e.g., agent 140 occludes theprojection in some way, such as blocking the projection with a limb, orplacing an object or other item in the path of projection 114).

As scene 106 also illustrates, textual information associated withvisual cue 116 may be projected, displaying the text “INCORRECT ITEM.”In some embodiments, the projection may be made based on a viewingperspective of agent 140. For instance, if agent 140 has moved locationswith respect to workstation surface 122 and item 150 a between scenes104 and 106, the projection may be adjusted so that the textualinformation is displayed at an orientation that may be read by agent140. For instance, agent 140 in scene 106 is now directly across fromitem 150 a (when agent 140 was further away and at an angle from item150 a in scene 104). From agent 140's viewing perspective, visual cue116 is oriented to be readable. In some embodiments, visual cue 116 mayprovide a user interface projected onto workstation surface 122, whichmay allow agent 140 to interact with the user interface via inputs onthe surface of 122, for example with a touch input onto the userinterface projection, visual indicator (e.g., laser pointer), or gesture(e.g., placing finger over a particular input control). Workstationvisual feedback system 110 may be configured to interpret imaging dataof the interaction of agent 140 with the projected user interface todetect input, selections, and other indications proffered via the userinterface.

Please note that previous descriptions are not intended to be limiting,but are merely provided as an example of a workstation, workstationsurface, and/or workstation visual feedback system. For example,different components of workstation visual feedback system 110 may bedifferently located. Other differences may include changes to thephysical layout or arrangement of workstation 120 and/or workstationsurface 122.

The workstation surface, for instance may be irregular, multi-level, ororiented in a different angle. In some embodiments, the workstationsurface may appear to be vertically oriented. Visual cues projected ontoworkstation surface 122 may also be different than illustrated in FIG.1.

This specification begins with a general description of a materialshandling facility implementing visual task feedback for workstationsimplemented as part of the materials handling facility. Differentitem-handling tasks or processes in the materials handling facility may,in some embodiments, implement visual task feedback for respectiveworkstations that implement these item-handling tasks or processes. Thenvarious examples of a visual feedback system for a workstation in amaterials handling facility are discussed, including differentcomponents/modules, or arrangements of components/module that may beemployed as part of providing visual cues. A number of different methodsand techniques to perform visual task feedback for workstations in amaterials handling facility are then discussed, some of which areillustrated in accompanying flowcharts. Finally, a description of anexample computing system upon which the various components, modules,systems, devices, and/or nodes may be implemented is provided. Variousexamples are provided throughout the specification.

FIG. 2 is a block diagram illustrating a logical representation of amaterials handling facility, according to some embodiments. In variousembodiments, a fulfillment network including multiple materials handlingfacilities (each of which may be configured in a manner similar to thatof materials handling facility 200) may be responsible for fulfillingmultiple orders, such as orders placed through an electronic commerce(“e-commerce”) portal.

In various embodiments, a materials handling facility may include one ormore facilities that process, store, and/or distribute units of itemsincluding but not limited to warehouses, distribution centers, hubs,fulfillment centers, nodes in a supply chain network, retailestablishments, shipping facilities, stock storage facilities, or anyother facility configured to process units of items. For example, FIG. 2may illustrate an order fulfillment center of a product distributor,according to some embodiments. Multiple customers 210 may submit orders220 to the product distributor through an e-commerce portal or otherelectronic marketplace, where each order 220 specifies one or more itemsfrom inventory 230 to be shipped to the customer that submitted theorder. To fulfill the customer orders 220, the one or more itemsspecified in each order may be retrieved, or picked, from inventory 230(which may also be referred to as stock storage) in the materialshandling facility, as indicated at 240. Picked items may be delivered orconveyed, if necessary, to one or more workstations in the materialshandling facility for sorting 250 into their respective orders, packing260, and finally shipping 270 to the customers 210.

Each item stored in inventory 230 may include an item identifier. Theterm item identifier may refer to a unique identifier associated witheach particular type of item carried in inventory 230 of a materialshandling facility 200. Multiple items of the same type of item may becarried in inventory 230. Typically, but not necessarily, each item istagged or otherwise marked with the item identifier. For example, unitsor collections of items in inventory may be marked or tagged with a barcode, Universal Product Code (UPC), Stock-Keeping Unit (SKU) code,serial number, and/or other designation (including proprietarydesignations) that may be used as item identifiers to facilitatematerials handling facility operations, including, but not limited to,stowing, rebinning, picking, sorting, packing and shipping. Thesedesignations, or codes, may identify items by type, and/or may identifyindividual items within a type of item.

Cases, boxes, bundles, or other collections of items may similarly bemarked or tagged with item identifiers. The items in a collection mayall be of the same type of item, for example a case of twelve items of aparticular item type, or may be a collection of one or more items ofeach of two or more heterogeneous item types. A collection of item(s)(e.g., a case containing twelve items, or a bundle containing one ormore items of each of two or more heterogeneous item types, such as aboxed or bundled set of three different books) may thus be considered ortreated as an “item” in the materials handling facility 200. Adesignation, or code, may thus also identify a collection of items as an“item” in the order fulfillment process. Thus, various processes, inaddition to sorting individual items, may also process a collection ofitems designated as an item. Therefore, the conveyance receptaclesdescribed herein may receive collections of items that are designated asan item as well as individual items.

As noted above, in various embodiments, picked items may be delivered toan induction workstation. Induction workstations may perform variousitem handling processes or tasks, in some embodiments. For instance, atthe induction station, individual items may be associated with andplaced into particular conveyance receptacles. The item may beassociated with the particular conveyance receptacle it is placed in. Insome embodiments, the association of an item with a particularconveyance receptacle may be performed by reading, scanning or otherwiseentering an item identifier associated with the item and a conveyancereceptacle identifier associated with the particular conveyancereceptacle into which the unit is placed. The item identifier andreceptacle identifier may be communicated to control system 202 of thematerials handling facility via wired and/or wireless communications.Each conveyance receptacle may include a unique conveyance receptacleidentifier that uniquely identifies the particular conveyance receptaclein the materials handling facility. The conveyance receptacle identifiermay, for example, be indicated by a bar code, Radio Frequency Identifier(RFID) device, or some other scannable or readable mechanism, mark, ortag attached to or integrated with the conveyance receptacle.

In some embodiments, the conveyance receptacles may then be routed toparticular destinations for the items contained within the receptaclesin accordance with the requests (orders) currently being processed. Forexample, sorting workstations may be implemented to sort items intorespective orders. Streams or batches of incoming items may arrive at asorting station and be grouped or placed in respective containers,modules, bins, slots, or other receptacles corresponding to a particularorder. In some embodiments, various item-handling tasks, similar tothose described above with induction may be performed, such as reading,scanning or otherwise entering an item identifier associated with theitem and a sorting receptacle identifier associated with the particularsorting receptacle into which the unit is placed. In some embodiments,robotic or automated transport systems may convey items in verticalstorage containers with multiple slots for storing items. Anitem-handling process may be taking items from or placing items into thevertically oriented storage containers.

An order may then proceed to a packing workstation, in some embodiments,to be packaged for shipping 270, under direction of a control system(e.g., control system 202). At each packing station, shipments of items(e.g., shipment) may be packed into a respective shipping container(e.g., corrugated box or other shipping container) for shipment from thematerials handling facility. Prepared shipments may be processed atshipping 270 and conveyed to a shipment carrier for delivery torespective customers. A picked, packed and shipped order does notnecessarily include all of the items ordered by the customer; anoutgoing shipment to a customer may include only a subset of the ordereditems available to ship at one time from an inventory storage location.

A materials handling facility may also include a receiving 280 operationfor receiving shipments 290 of stock (e.g., units of inventory items)from one or more sources (e.g., vendors) and for moving or “stowing” thereceived stock into stock storage (e.g., inventory 230). The receiving280 operation may also receive and process returned purchased or renteditems or orders from customers. At least some of these items aretypically returned to inventory 230. The various operations of amaterials handling facility may be located in one building or facility,or alternatively may be spread or subdivided across two or morebuildings or facilities. In various instances, it should be understoodthat references to elements, units, items, processes (or anything else)as being located within materials handling facility 200 may easily beextended to encompass elements, units, items, processes (or anythingelse) proximate to but not physically located within materials handlingfacility. For example, various elements, units, items, or processes (oranything else) may be implemented outside of the materials handlingfacility, according to some embodiments.

In various embodiments, shipments of one or more items at shipping 270may be transferred to one or more shipment carrier network(s). Eachshipment carrier's network may include one or more distributionfacilities for storing items as well as vehicles for conveying shipmentsfrom such distribution facilities and/or materials handling facilities(such as materials handling facility 200) to various destinations (e.g.,customer specified destinations).

A materials handling facility may include a control system 202 which mayinclude, but is not limited to, one or more computer systems, one ormore data storage devices, one or more wired and/or wireless networks,control system software (programs, modules, drivers, user interfaces,etc.), and one or more hand-held, mobile and/or fixed readers, scannersor scanning devices that may be able to scan, receive, or otherwisedetect the marks or tags (e.g., bar codes, radio frequencyidentification (RFID) tags, etc.) on individual items or collections ofitems (e.g., cases) and communicate with a control station or stationsof the control system to, for example, determine and record the itemand/or item type of the items. The hand-held, mobile and/or fixedreaders, scanners or scanning devices may also be able to scan, receive,or otherwise detect the marks or tags (e.g., bar codes, radio frequencyidentification (RFID) tags, etc.) attached to or integrated with theconveyance receptacles. An exemplary computer system that may be used ina control system 202 is illustrated in FIG. 11.

Control system 202 may manage or direct the performance of variousoperations, processes, or item-handling tasks in materials handlingfacility 200 to efficiently utilize resources of the materials handlingfacility without exceeding known constraints of the materials handlingfacility. For example, control system 202 may manage the flow of itemsthroughout materials handling facility 200. In some embodiments, controlsystem 202 may direct or monitor the performance of various othersystems in materials handling facility 200, such as a workstation visualfeedback system described below.

FIG. 3 is a block diagram illustrating a logical illustration of aworkstation visual feedback system, according to some embodiments.Workstation visual feedback system(s) 300 may implement differentrespective sensors to capture data with regard to a workstation surface,such as imaging sensor(s) 322, depth sensor(s) 324, and scale(s) 326that may be communicated to control system 202 through various wired orwireless communication technologies. Control system 202 may beconfigured to make various determinations as to the current state of anitem-handling task performed on a workstation surface and/or performdifferent item recognition techniques to identify items on theworkstation surface based on the captured data and information obtainedfrom data store 340. Agent gestures, or other changes in the workstationenvironment (e.g., area around the workstation surface) may also berecognized. Visual task cues to project may be selected based on theidentified items and/or state of the item-handling task. Visualprojector(s) 334 may then project the selected visual task cues onto theworkstation surface as directed by control system 202. Visual projectorsmay be any form of light-based projection that projects image data(e.g., a video signal) onto the workstation surface (e.g., CRTprojector, liquid crystal display (LCD) projector, digital lightprocessing (DLP) projectors, or any other projection technology). Insome embodiments, visual projectors may also include other displaytechnologies which may create the appearance of projections on theworkstation surface from the viewing perspective of an agent (e.g.,where the workstation surface itself incorporates a surface display, orwearable display technology such as a heads up display (HUD)). Controlsystem 202 may again communicate with agent trackers 332, visualprojector(s) 334, and audio 336 over wired or wireless communicationtechnologies.

In various embodiments, control system 202 may implement anitem/task/input recognition module 320 to perform itemidentification/recognition, task state recognition, or gesturerecognition (e.g., for a user input such as may be projected onto aworkstation surface as part of workstation visual feedback system(s) 300or task performance analysis). For example, item/task/input recognitionmodule 320 may be configured to perform one or more different imagerecognition, computer vision, or object recognition techniques, such asedge matching, feature detection, or template matching, to identifyitems located on a workstation surface by evaluating image data of theworkstation surface received from imaging sensor(s) 322 and/or depthsensor(s) 324, as well as other information from sensors such asscale(s) 326. In some embodiments, items/task/input recognition module320 may be configured to identify barcodes, labels, or other itemidentifiers located on items. Data store 340 may maintain descriptiveinformation about items carried in a materials handling facility (e.g.,images, physical characteristics, such as height, length, width, weight,or item identifier numbers). Thus, depth information obtained from depthsensor(s) 324 corresponding to imaging data from imaging sensor(s) 322may be used to determine geometric information (e.g., the length, width,and/or height of items), for example. Scale(s) 326 may provide weightinformation, either of individual items on a workstation surface, or thetotal weight of items on a workstation surface as a whole. In someembodiments, item/task/gesture recognition module 320 may also beconfigured to perform various video tracking techniques to track anddetermine the current state of an item-handling task being performed byan agent at a workstation. Data store 340 may also include informationpertaining to the performance of item-handling tasks in order todetermine the current state of the item-handling task. Similarly,item/task/gesture recognition module 320 may evaluate the image data torecognize gestures associated with the performance of a task.

Visual cue manager 330 may be implemented at control system 202 toselect, generate, modify, and otherwise manage visual task cues. Visualcue manager may receive items identified by item task input manager 320and make a selection of a visual task cue to provide feedback at theworkstation surface. Data store 340 may be accessed to identify alisting of items associated with the particular item-handling task beingperformed (e.g., items associated with a particular order or productassembly task). Missing items, incorrect items, or unrecognizable itemsmay be determined. FIG. 8, discussed below, provides examples of variousdifferent techniques that visual cue manager 330 may implement to selecta visual task cue. Once selected, visual task cue manager 330 may directvisual projector(s) 334 to project the selected visual task cue. In someembodiments, agent tracker(s) 332 may provide location information ofthe agent to visual cue manager 330. Visual cue manager 330 may alsodetermine appropriate visual cues for the current state of task that isdetermined by item/task/gesture recognition module 320 or for recognizedgestures by item/task/gesture recognition module 320.

Visual cue manager 330 may determine a viewing perspective for the agentaccording to which currently projected visual task cues may be updatedto be projected in accordance with the perspective, or a new visual taskcue to be projected in accordance with the determined perspective.Visual cue manager may implement the various techniques described belowwith regard to FIG. 10 to project visual task cues according to agentperspective. Visual cue manager 330 may also modify or update projectedvisual task cues according to detected surface changes (which althoughnot illustrated may be determined based on geometric or depthinformation from depth sensors 324) or other workstation environmentchanges (e.g., object or agent obstruction within the workspace, such asthe area between projectors and the workstation surface). FIG. 9,described below, provides many examples of techniques which visual cuemanager 330 may implement to adjust projected visual task cues. In someembodiments, visual cue manager 330 may also be configured to interpretinput for user interfaces projected as visual cues onto a workstationsurface, such as discussed below with regard to FIG. 8.

Workstation visual feedback system 300 may, in some embodiments, beintegrated with a materials handling facility control system, such asmaterials handling facility control system 202 described in FIG. 2.Different ones of the various workstations (e.g., receiving, stowing,inducting, sorting, or packing) may implement a respective workstationvisual feedback system 300, in some embodiments. In some embodiments,visual workstation feedback system(s) 300 may be implemented atparticular types of workstations (e.g., implemented at packingworkstations).

FIG. 4 is a diagram illustrating a workstation implementing aworkstation visual feedback system, according to some embodiments.Workstation visual feedback system 400 may be implemented as fixed orworkstation-based feedback system. In some embodiments, a control systemmay be located at or near the workstation, or may be separately locatedand in communication with the other components illustrated in FIG. 4. Invarious embodiments, one or more image sensors 420 may be located insuch a way as to have a view of workstation surface 402. For example, insome embodiments, image sensors 420 (as well as the other components)may be located above workstation surface 402. For workstation surfaces402 with a different orientation (e.g., a vertical orientation), theimaging sensors may also be located in such a way as to have a view ofthe workstation surface 402 that may be different than above theworkstation surface.

Image sensors 420 may be one or more digital still image and/or videocameras, configured to capture and/or monitor workstation surface 402.Image data captured by image sensors 420 may be formatted according toone or more different formats and/or captured at one or more differentimage resolutions. In some embodiments, multiple image sensors 420 maycapture data of the workstation surface, with each sensor responsiblefor some or all of workstation surface 402. In some embodiments, imagesensors 420 may be located in different positions with different vantagepoints of workstation surface 420. One or more depth sensors 430 may beimplemented, in some embodiments. For example, time-of-flight cameras,plenoptic or light-field cameras, stereo cameras implementingtriangulation, or any other range imaging sensor that may provide depthinformation (or geometric information) between the depth sensor 430 andthe workstation surface may be implemented. In some embodiments, imagingsensors 420 may include the capabilities of depth sensor 430, and thusone device may capture both image data and depth information. In someembodiments, the imaging sensors 420 may use various techniques forconstructing geometric information of the workstation surface and/or theenvironment surrounding the workstation surface (e.g., using multipleimages or other stereo reconstruction techniques). In some embodiments,a scale 410 may be implemented to obtain weight information for itemslocated on workstation surface 402.

One or more projectors 440 may be implemented to project visual taskcues onto workstation surface 402. These projectors may simply producelaser or other simple pointer imagery, or may be configured to projectvarious colors, shapes, sizes, or animations as the visual task cues.Audio 450 may be one or more audio speakers configured to provide audiocues in addition to and corresponding to projected visual task cues.Agent tracking sensor 460 may be a sensor configured to determine agentlocation information, such as the current location of an agent's body,head, eyes, or gaze.

Although illustrated in FIG. 4 as a workstation-based visual feedbacksystem, in some embodiments, one or more components may be implementedin another location. For example, various forms of wearable technology,such as a heads-up display (HUD), may be worn by an agent andprojections may be configured to display on the wearable technology insuch a way that it appears that the visual task cue is projected ontothe workstation surface from the perspective of the agent wearing thewearable technology. In some embodiments, a surface-based projector maybe implemented such that the very surface of the workstation may beconfigured to display visual task cues onto itself

FIGS. 5A-5F are diagrams illustrating different visual cues projectedonto a pack workstation surface, according to some embodiments. Packstation 500 is configured to allow an agent, agent 525, to perform anitem-handling task, a packing process to pack items 570 together into ashipping container. Items 570 are located on pack station surface 540for performance of the packing process. Visual feedback system 510 (or aportion thereof) including various imaging and/or depth sensors, as wellas projectors, audio speakers, and/or agent tracking sensors may beimplemented in a fixed position above workstation surface 540. Agent 525packs items into shipping containers and places the packed container 590onto conveyer mechanism 580 to be conveyed to another location in amaterials handling facility. Container inventory 530 may includeshipping containers of varying sizes and/or properties that may be usedto perform the packing process.

In FIG. 5A, visual feedback system 510 has projected an item-specificvisual cue 550 a onto one of items 570 (as illustrated by the shadedarea). This type of visual cue may convey one or more indications toagent 525. For instance, the item may not be recognized by visualfeedback system and require manual entry, or the item may be the wrongitem to pack. In some embodiments, the item-specific visual cue 550 amay indicate that the highlighted item is to be the first packed into ashipping container.

FIG. 5B illustrates another type of visual task cue, a surface generalcue 550 b. For instance, in some embodiments, a visual task cue may beprojected onto a general or entire area of pack station surface 540.This may indicate several things. For example, the surface general cuemay provide a halt or proceed indication (e.g., by projecting a red orgreen object onto the workstation surface) corresponding to a currentstate of the task (e.g., not all items are packed). The surface generalcue 550 b may indicate items remaining to be identified, packed, or someother operation performed upon, which after completion the portion ofthe visual cue highlighting that item may be removed until tasks for theall of the items are complete and no item remains highlighted.

FIG. 5C illustrates another type of visual task cue, a surface specifictask cue 550 c. A surface specific task cue may indicate that aparticular item is missing, or should be located where the visual taskcue 550 c is located. FIG. 5D illustrates another type of visual taskcue, a text indication/user interface visual cue 550 d. As noted above,textual information describing next steps, current errors, otheritem-task handling instructions may be projected onto station surface540. In this way, agent 525 need not look away to a different monitor,book, or other information source to obtain information. Instead, agent525 may maintain focus on the area in which agent 525 is currentlyworking. User interface cue 550 d may be a user interface which allowsagent to provide input, such as by touching or hovering over one or moreinput objects within the projected user interface 550 d. Again, agent525 need not look away or interact with a different display system toprovide input.

FIG. 5E illustrates another type of visual task cue, an agent-localizedvisual cue 550 e. An agent-localized visual cue may project the task cueonto the agent (e.g., agent 525's hand in FIG. 5E). Agent may beconsidered part of the workstation surface as determined from visualfeedback system 510, and thus projecting the visual task cue onto agent525 may be (from the perspective of visual feedback system 510)considered the same or a part of the workstation purpose. FIG. 5Fillustrates another type of visual task cue, an object-specificinstruction visual cue 550 f. An object-specific instruction visual cue550 f may be a projection of instructions or visual indication directlyonto the object or portion of the object to which they pertain. Forexample, in an assembly item-handling task performed at a workstation,the object-specific instruction may highlight the portion of two itemsthat are to be connected together.

The examples of visual task feedback for workstations in materialshandling facilities discussed above with regard to FIGS. 2-5F have beengiven in regard to a fixed or workstation-based visual task feedbacksystem in a materials handling facility. Various other types orconfigurations of visual task feedback for workstations in materialshandling facilities may implement these techniques. For example, ratherthan a control system like control system 202 discussed above, anindividual control system may be implemented for an individualworkstation. In some embodiments, computing systems or devices may bephysically located near a workstation, or configured to communicate withsensors and/or projectors near a workstation from a central location.Different configurations or types of sensors for image, geometricinformation, and/or weight may be implemented, as well as differentconfigurations and types of projectors. FIG. 6 is a high-level flowchartillustrating various methods and techniques for performing visualfeedback for workstations in materials handling facilities, according tosome embodiments. The various embodiments described above, as well asother systems or configurations may implement the following techniques.

As indicated at 610, image data of a workstation surface at aworkstation in a materials handling facility may be obtained via one ormore image sensors, in various embodiments. Image data may includevarious high resolution images or video formatted, compressed, orcommunicated in one or more formats. Real-time or near real-time imagedata may be captured. Geometric information may also be obtained via oneor more image and/or depth sensors (which may be one or more of theimage sensors configured to determine depth or geometric information).Geometric information may also be reconstructed based on image datacaptured for the workstation. Depth information may also be obtained andcorrespond to the image data of the workstation surface such that thedistance between the depth sensor and items on the workstation surfaceas discerned from the image data may be known. For example, the side ofa box facing a depth sensor may be a certain distance away from thedepth sensor, while the bare surface of the workstation sensor may be agreater distance away from the depth sensor, thus indicating a heightdimension for the box. Image data and/or corresponding geometricinformation may be specific to a portion of the workstation surface, theentire workstation surface, and/or the environment or area surroundingthe workstation surface visible to the imaging sensors. For example,imaging and/or depth sensors may only monitor a center portion of aworkstation surface, a raised portion of a workstation surface, or alarger area including the workstation surface and its surroundingenvironment.

Weight or mass information about items on the workstation surface mayalso be obtained via one or more scales. For example, a scale may beembedded in workstation surface upon which items are placed (or possiblyinitially placed). Weight information may be obtained that is specificto a particular item on the workstation surface or all items on theworkstation surface.

As indicated at 620, the image data may be evaluated with regard to aperformance of an item-handling task at the workstation. For example, insome embodiments one or more items located on the workstation surfaceassociated with an item-handling task performed at the workstation maybe identified based, at least in part on the image data. Many differentitem recognition techniques may be performed, including, but not limitedto edge matching, feature detection, or template matching. In someembodiments, image data may be analyzed to locate item identifiers(e.g., barcodes or numbers) on items on the workstation surface. Othertypes of information corresponding to or used in combination with imagedata may be used to identify items, such as depth informationcorresponding to the image data of the workstation surface and/or weightinformation indicating the weight of one or more items on theworkstation surface. FIG. 7 described in more detail below discussesmultiple ways in which items located on the surface of the workstationmay be identified.

In some embodiments, an item may be identified as an item (as opposed toa feature of the workstation surface) but may not be recognizable as tothe specific type or unit. For instance, 3 books may be located on theworkstation surface. Based on a comparative analysis, 2 of 3 books maybe identified and recognized as Book A and Book B (along with respectiveitem information describing the item such as publisher, author, weight,dimensions, etc.). The remaining book however may be unrecognizable,either as the specific book title, or as a book at all. Instead, theremaining book may be identified as an unrecognizable item. As notedbelow, a visual task feedback cue may be selected to indicate manualentry of an item identifier of an unrecognizable item. Then, in theabove example, the book may be identified as Book C (along with itsrespective information). In some embodiments, the items located on theworkstation surface may be identified according to the item recognitiontechniques described herein, and the item-handling task may be performedwithout manual scan or entry of the items for recognition. This may leadto more efficient performance of the item handling task as a manualentry, scan, or other step to recognize items may not need to becompleted.

In some embodiments, the image data may be evaluated in order todetermine a current state of the performance of the item-handling task.For example, the percentage of completion or progress of theitem-handling task may be identified. A particular stage of theitem-handling task (e.g., preparation, assembly, or packing) may beidentified. In some embodiments, the current state of the item-handlingtask may correspond with the performance of certain agent movements orgestures recognizable by evaluating the image data.

In some embodiments, the image data may be evaluated to recognize agentgestures with regard to the performance of the item-handling task at theworkstation. For instance, certain packing motions may be recognized(e.g., obtaining a shipping container). In some embodiments, thecorrectness of the motions may be determined, for instance selecting thewrong shipping container or performing the gestures in an incorrect orinefficient order or manner.

As indicated at 630, a determination as to whether to project one ormore visual task cues onto the workstation surface may be made based, atleast in part on the evaluation of the image data, in some embodiments.Many different types of visual task cues may be determined. For example,visual task cues may indicate that specific actions are to be performed,or that erroneous items or actions have been obtained/performed. Invarious embodiments, the one or more visual task cues may indicate acurrent state of the item-handling task associated with the one or moreidentified items. For instance, if an item is missing or incorrect, acorresponding visual task cue indicating such a missing or incorrectitem may indicate that the state of the item-handling task isincomplete. Similarly, visual task cues to manually identify items orprovide input to a projected user interface may also indicate a currentstate of the item-handling task, such as steps or actions remaining tobe performed. Indications of completion are another type of visual taskcues that may be determined based on the identified items. FIG. 8discussed in further detail below describes many techniques fordetermining and/or selecting visual task cues according to someembodiments. If, as indicated by the negative exit from 630, no visualtask cues are determined, more image data may be again obtained and thetechnique may begin again. In this way, visual task cues may be provideddynamically during the performance of an item-handling task, respondingto changes on the surface of the workstation.

As indicated at 640, if one or more visual task cues are determined,then projection of the visual task cues onto the surface of theworkstation may be directed, in some embodiments. The visual task cuesmay, in some embodiments, be projected according to the image data ofthe workstation surface. For instance, the visual task cues may beprojected onto particular locations on the workstation surface asidentified via the image data. In another example, projection of thevisual task cues may account for the features of the workstation surfaceonto which they are projected as determined according to the image dataand/or geometric information, in some embodiments. For example, theprojection of a visual task cue may be projected to accurately cover oneor more surfaces of an item (e.g., in effect painting or highlightingthe item) without being projected on other items and/or the surface ofthe workstation based on item dimensions calculated from item imageand/or depth information. In some embodiments, a visual task cue may beselected, generated, and/or projected to neutralize portions of theworkstation surface (e.g., by graying out other items).

As indicated by the positive exit from 650, if the item handling task iscomplete, the feedback may end (for this particular performance of theitem handling task). If, however, the item-handling task is not complete(as indicated by the negative exit from 650), then the previouslydescribed elements may be repeated.

FIG. 7 is a high-level flowchart illustrating various methods andtechniques for performing item recognition for items located on aworkstation surface, according to some embodiments. As indicated at 710,image data of the workstation surface may be captured via imagingsensors. Imaging sensors may view some or all of a workstation surfaceand may be provide varying kinds of image data, such as video or stillimages, according to multiple different formats and resolutions. In someembodiments, depth information corresponding to the image data of theworkstation surface may be captured, as indicated at 720. Geometricinformation may be useful to perform various different types of itemrecognition techniques. For instance, in some embodiments, geometricinformation along with image data may be used to calculate one or moreitem dimensions for items located on the workstation surface. Geometricinformation may also be used to more accurately understand the shape,outline, or profile of an item which may not be intelligible based onthe image data alone. As indicated at 730, in some embodiments, weightinformation may also be obtained from one or more scales.

In some embodiments, a data store, such as data store 340 describedabove with regard to FIG. 3, may be maintained for items carried,stored, managed, or processed in a materials handling facility and maybe accessed, as indicated at 740. The data store may maintaindescriptive information associated with particular items in thematerials handling facility that may be used to identify and recognizeitems. Item identifiers, item dimensions, shapes, colors, image data,weight, or any other form of descriptive information may be maintainedin the data store. Once accessed, the image data, geometric information,and/or weight information may be compared with the item descriptiveinformation in the data store to identify items on the workstationsurface. For example, high resolution image data may capture itemidentifiers (e.g., bar codes) on the items which may be compared withitem identifiers stored in the data store. Other recognition techniquescomparing stored image data maintained for different items in the datastore with captured image data of the workstation surface (e.g., edgedetection comparisons, histogram comparisons, or other objectrecognition techniques) may be performed. In some embodiments, geometricinformation and/or weight information may be used for secondary oralternative recognition techniques. For example, if an item imagecomparison makes a recognition decision for an item below a certainconfidence threshold, then geometric and/or weight information may beused to determine dimensions of an item on the workstation surfaceand/or the weight of the item on the workstation surface and compare itwith dimensions and/or weight for the possible item maintained in thedata store to ensure that one or both alternative recognition techniquesagree.

As noted above with regard to FIG. 6, visual task cues may be determinedor selected that indicate a state of the item handling task that isbeing performed. Different states of the item-handling task maynecessitate different information or feedback be presented to an agentperforming the item-handling task at the workstation surface. FIG. 8 isa high-level flowchart illustrating various methods and techniques forselecting a visual task cue to be projected onto a workstation surface,according to some embodiments.

As indicated at 810, the image data of the workstation surface may beanalyzed, such as performing the comparative analysis described abovewith regard to FIG. 7 and/or the other techniques discussed above withregard to FIG. 6. Items located on the workstation surface may beidentified based, at least in part, on the evaluation of the image data,the current state of the item-handling task may be determined, or agentgestures may be recognized. One determination that may be made iswhether an item located on the workstation surface is recognizable, asindicated at 820. For instance, the image data may be able distinguishthat 6 items are located on a workstation surface. By performing acomparative analysis, 5 of the 6 items may be recognized (e.g., havetheir corresponding item information located in an item data store bydetermining the item's identity). The remaining item, though it may bedistinguishable from the other items and the workstation surface, maynot be able to recognized using the various identification techniques.For instance, the item may be positioned on the workstation surface toreduce or hide distinguishing features (e.g., may hide a barcode orother identifier that may be recognizable). As indicated by the positiveexit from 820, if the item is unrecognized, a visual task cue may beselected to indicate manual item identification is to be performed 822,such as by scanning the item identifier or entering an item identifiervia an input device (e.g., scanner or keyboard). A visual task cueindicating this may be implemented in a variety of ways, such ashighlighting the unrecognizable item in a particular color or projectingan arrow and/or instructions regarding the item.

As indicated at 830, in some embodiments, a determination may be made asto whether a particular item is missing from the workstation surfacebased on the image data. The item-handling task being performed maycorrespond to a particular one or more items. For example, a workstationthat performs a packing process may have a set of items associated withan order that are to be packed together in a shipping container.Similarly, in another example, a workstation that performs some assemblyprocess may have a list of items (e.g., parts) that may be required tocomplete the assembly process. When item identification is performed,the identified items may be checked or compared against the particularitem(s) corresponding to the item-handling task. If an item is missing,as indicated by the positive exit from 830, a visual task cue may beselected that indicates that an item is missing 832. For example, ageneral warning or indication, such as a visual task cue that paints abright color across a substantial portion of the workstation surface maybe projected. Or, in another example, an empty space on the workstationsurface may be highlighted and may include textual informationdescribing the missing item.

In some embodiments, a missing item may be an item that is determinedfor the item-handling task at the time of performing the item-handlingtask. For example, in some embodiments, the workstation may be a packworkstation, such as described above with regard to FIGS. 2-5F, where anitem-handling task to pack together items into a shipping container maybe performed. A determination may be made based on image, depth, weight,and/or other data used to identify the items (as discussed above withregard to FIGS. 6 and 7) to select (or recommend) a shipping container(e.g., corrugated box or other shipping container). This determinationmay be made in real or near-real time. In some embodiments, theselection of the shipping container may replace a previously selected(or recommended) shipping container for the item(s). Thus, in someembodiments, the visual task cue indicating the missing item mayidentify the selected shipping container as the missing item.

As indicated at 840, a determination may be made that a particular itemis incorrect (or extraneous) for performing the item-handling task.Similar to 830 above, the item-handling task being performed maycorrespond to a particular one or more items. When item identificationis performed, the identified items may be checked or compared againstthe particular item(s) corresponding to the item-handling task. If anitem is incorrect (e.g., not associated with the task or necessary toperform the task), as indicated by the positive exit from 840, then avisual task cue indicating an incorrect item 842 may be selected. Forexample, the incorrect item may be highlighted or painted a particularcolor (e.g., red) by the projected visual task cue.

In some embodiments, visual task cues may be selected that indicatevarious levels of completion of an item-handling task. For example, asindicated at 850, in some embodiments, a determination may be made thatall items are complete (or present) for the performance of the itemhandling task, and a corresponding visual task cue may be selectedindicating that all items are complete (or present) for the performanceof the item handling task. Consider again the pack workstation example,if all of the items associated with the order to be packed in a shippingcontainer are identified, then a visual cue may be selected andprojected that indicates that the remaining portion of the task (packingthe items into the shipping container) may be completed. Some visualtask indicators may indicate that an item-handling task is not yetcomplete. For example, in some embodiments, the image data of theworkstation surface may be analyzed to determine whether each step of anitem-handling task is complete. If a step is skipped, the visual taskcue may indicate that the item-handling task is incomplete (and mayprovide an indication as to the skipped step).

Please note that visual task cues discussed above with regard to FIG. 8are not intended to be limiting, as many other different visual taskcues may be selected for projection onto the workstation surface.Additionally, combinations of different visual task cues may also beprojected. Visual task cues may be projected at the same or differenttimes. In some embodiments, visual task cues may be projected in such away as to indicate an order or sequence of steps or operations to beperformed as part of the item handling task. For example, in an assemblyitem-handling task, a visual task cue indicating the next item to beused for assembly may be highlighted. Similarly, in another example,packing order may be determined for items to be placed into a shippingcontainer. The visual task cues may be projected onto the workstationsurface in order to indicate the packing order. For instance, each nextitem to place in the shipping container may be illuminated.

Another example of a visual task cue may be selecting a visual task cuethat operates as a user interface. For instance, in some embodiments, auser interface may be projected onto the workstation surface (e.g.,asking for confirmation or indications of certain task performancedecisions or actions taken by an agent). Imaging sensors and/or depthsensors may be used to recognize agent gestures as input action to theuser interface such as by touching or placing an object over or above aninput in the user interface, which may then be recognized via image dataand/or depth information as a selection or input.

Item-handling processes may entail many changes to or interactions withthe workstation surface or environment. New items may be added or workmay be performed on the surface (e.g., packing, assembling, or othermanual techniques where agents and/or tools operated by agents may enterthe workstation surface space). Changes to the workstation surface maydistort or modify visual task cues projected onto the workstationsurface. FIG. 9 is a high-level flowchart illustrating various methodsand techniques for adjusting projected visual task cues based onadditional image and depth information for a workstation surface,according to some embodiments. As indicated at 910, a visual task cuemay be projected onto a workstation surface. For instance, a particularitem or items may be highlighted, or instructions or text may bedisplayed. Based, at least in part, on image data and correspondingdepth information of the workstation surface, a surface change may bedetected, as indicated at 920, in some embodiments.

A surface change may, in various embodiments, be a change to the depthinformation corresponding to the surface of a workstation as perceivedby a depth sensor monitoring a workstation surface. As noted above,geometric information for a workstation surface may indicate thedistance between the depth sensor and the workstation surface. Objectson the workstation surface may decrease the distance between theworkstation surface and the depth sensor by appearing to raise theheight of the workstation surface (from the perspective of the depthsensor) to the height of the object intervening between the surface ofthe workstation and the depth sensor (e.g., the top of a box sitting ona workstation surface is perceived to be a higher elevation than thebare workstation surface on which it rests). Thus, a surface change maybe the addition of a new item onto the workstation surface (e.g., a box,tool, or item to be included in the item handling task that changes thedepth for the surface) or the movement of an already present item to adifferent location on the workstation surface. Stacking an item or boxon top of another item is another example surface change that may bedetected. More generally, any object that intervenes between or occludesa depth sensor and a workstation surface (e.g., an agent's limb, such asa hand, arm, etc.), even if only temporarily, may be a detectablesurface change. A surface change may also be a detected change in theenvironment of the workstation, such as a change in the lighting of theworkstation.

As indicated at 930, in response to detecting the surface change of theworkstation surface, one or more projection modifications may bedetermined, in some embodiments. For example, the location, size, shape,and/or coloration of the visual task cue may be adjusted. If an item isflipped or rotated differently on a workstation surface, the shape ofthe visual task cue that corresponds to the shape of the item may becorrespondingly adjusted, for instance. In some embodiments, one or moreother projection devices may project the visual task cue from adifferent vantage point, and thus modifications to the visual task cueto be properly displayed when projected from the different vantage pointmay be made.

As indicated at 940, the visual task cue projected onto the workstationsurface may be updated according to the projection modifications. Forexample, the adjustments to the projected location, size, shape, and/orcoloration of the visual task cue may be made to shift, resize, reshape,and/or recolor the visual task cue. The method described above withregard to FIG. 9 may be performed dynamically. For example, imagingsensors and/or depth sensors may monitor a workstation surface while acontrol system may dynamically detect the surface change, determinemodifications, and update projected visual tasks cues for multipledetected surface changes in or near-real time. If, for instance, anagent's hand blocks the projection of a visual task cue while over theworkstation surface, one or more other projectors may project the visualtask cue from a vantage point that would not be occluded by the agent'shand. When the agent's hand is removed from over the workstationsurface, the visual task cue may be projected again according to itsoriginally determined location, size, or other characteristic beforemodification.

While performing item-handling tasks, agents may physically relocate todifferent portions of a workstation surface and/or view differentportions of a workstation surface differently. With an agent's view ofthe workstation surface changing due to these differing views orlocations, visual task cues projected onto a workstation surface may notbe optimally displayed, understandable, or even visible at all. FIG. 10is a high-level flowchart illustrating various methods and techniquesfor projecting a visual task cue based on a viewing perspective of anagent at a workstation, according to some embodiments.

As indicated at 1010, location information for an agent performing anitem-handling task at a workstation in a materials handling facility maybe received. Various types of location information for an agent may bereceived via agent tracking sensors. For example, in some embodimentseye tracking or gaze tracking techniques (e.g., using a remote eyetracking device) may be implemented to determine the location on theworkstation surface that an agent is currently viewing to perform a task(e.g., looking a particular item), and/or a particular angle from whichthe agent is viewing the workstation surface. Similarly, in someembodiments, video tracking or other location tracking techniques thatmay be implemented to determine an agent's location relative to aworkstation surface.

As indicated at 1020, based, at least in part, on the received locationinformation, a viewing perspective for the agent may be determined. Forexample, a determination may be made as to an agent's field of view. If,for instance, an agent is closely viewing a particular item on aworkstation surface, the agent's field of view may be limited as to aportion of the workstation surface. This portion of the workstationsurface may be geographically determined as a viewing perspective forthe agent. Other techniques, such as determining the location and/ordirection of an agent's head relative to the workstation surface, inorder to determine a viewing perspective, may also be used.

In various embodiments, visual task cues may be projected onto theworkstation surface according to the viewing perspective for the agent,as indicated at 1030. In some embodiments, a default version of a visualtask cue (e.g., image or animation) may be implemented. The defaultversion of the visual task cue may then be modified according to theviewing perspective of the agent, such that from the agent's perspectivethe projected visual task cue appears to be the same as the defaultversion of the visual task cue. Thus, the viewing perspective of anagent may also be used to project visual task cues in a way that isintelligible to the agent. For instance, if textual information isprojected as part of a visual task cue, the viewing perspective for theagent may be used to determine an orientation of the textual informationthat is readable (e.g., not upside-down) from the perspective of theagent. Other visual task cues may rely upon the viewing perspective ofthe agent to achieve a particular effect, such as an anamorphic visualtask cue (which may appear 3-dimensional to the agent when viewed from aparticular vantage point, such as an anamorphosis). Other examples thataccount for determined agent viewing perspectives when projecting visualtask cues may include projection of visual task cues that would not bewithin an field of view for an agent currently that may be modified tobe included in the field of view for the agent (e.g., if a shaded orbanded area around the item is the visual task cue, the size of theshaded or banded area may be increased). In another example, additionalvisual task cues may be displayed to signal and/or direct the gaze of anagent to the visual task cue, such as by projecting an arrow or otherdirectional symbol within the field of view of the agent, or projectinga large scale flash or other signal that the agent can view andinterpret as a signal to look for the visual task cue. Similar to FIG. 9discussed above, changes in location information may result indetermining updated viewing perspectives for an agent, with theprojection of currently projected visual task cues updated to correspondto the new viewing perspective, in some embodiments.

The methods described herein may in various embodiments be implementedby any combination of hardware and software. For example, in oneembodiment, the methods may be implemented by a computer system (e.g., acomputer system as in FIG. 11) that includes one or more processorsexecuting program instructions stored on a computer-readable storagemedium coupled to the processors. The program instructions may beconfigured to implement the functionality described herein (e.g., thefunctionality of various control systems, managers and/or othercomponents, such as those that implement the workstation visual feedbacksystem described herein). The various methods as illustrated in thefigures and described herein represent example embodiments of methods.The order of any method may be changed, and various elements may beadded, reordered, combined, omitted, modified, etc.

Embodiments of visual task feedback for workstations in materialshandling facilities as described herein may be executed on one or morecomputer systems, which may interact with various other devices. FIG. 11is a block diagram illustrating an example computer system, according tovarious embodiments. For example, computer system 2000 may be configuredto implement a control system in different embodiments. Computer system2000 may be any of various types of devices, including, but not limitedto, a personal computer system, desktop computer, laptop or notebookcomputer, mainframe computer system, handheld computer, workstation,network computer, a consumer device, application server, storage device,telephone, mobile telephone, or in general any type of computing device.

Computer system 2000 includes one or more processors 2010 (any of whichmay include multiple cores, which may be single or multi-threaded)coupled to a system memory 2020 via an input/output (I/O) interface2030. Computer system 2000 further includes a network interface 2040coupled to I/O interface 2030. In various embodiments, computer system2000 may be a uniprocessor system including one processor 2010, or amultiprocessor system including several processors 2010 (e.g., two,four, eight, or another suitable number). Processors 2010 may be anysuitable processors capable of executing instructions. For example, invarious embodiments, processors 2010 may be general-purpose or embeddedprocessors implementing any of a variety of instruction setarchitectures (ISAs), such as the x86, PowerPC, SPARC, or MIPS ISAs, orany other suitable ISA. In multiprocessor systems, each of processors2010 may commonly, but not necessarily, implement the same ISA. Thecomputer system 2000 also includes one or more network communicationdevices (e.g., network interface 2040) for communicating with othersystems and/or components over a communications network (e.g. Internet,LAN, etc.).

In the illustrated embodiment, computer system 2000 also includes one ormore persistent storage devices 2060 and/or one or more I/O devices2080. In various embodiments, persistent storage devices 2060 maycorrespond to disk drives, tape drives, solid state memory, other massstorage devices, block-based storage devices, or any other persistentstorage device. Computer system 2000 (or a distributed application oroperating system operating thereon) may store instructions and/or datain persistent storage devices 2060, as desired, and may retrieve thestored instruction and/or data as needed.

Computer system 2000 includes one or more system memories 2020 that areconfigured to store instructions and data accessible by processor(s)2010. In various embodiments, system memories 2020 may be implementedusing any suitable memory technology, (e.g., one or more of cache,static random access memory (SRAM), DRAM, RDRAM, EDO RAM, DDR 10 RAM,synchronous dynamic RAM (SDRAM), Rambus RAM, EEPROM,non-volatile/Flash-type memory, or any other type of memory). Systemmemory 2020 may contain program instructions 2025 that are executable byprocessor(s) 2010 to implement the methods and techniques describedherein. In various embodiments, program instructions 2025 may be encodedin platform native binary, any interpreted language such as Java™byte-code, or in any other language such as C/C++, Java™, etc., or inany combination thereof. For example, in the illustrated embodiment,program instructions 2025 include program instructions executable toimplement the functionality of a control system, in differentembodiments.

In some embodiments, program instructions 2025 may include instructionsexecutable to implement an operating system (not shown), which may beany of various operating systems, such as UNIX, LINUX, Solaris™, MacOS™,Windows™, etc. Any or all of program instructions 2025 may be providedas a computer program product, or software, that may include anon-transitory computer-readable storage medium having stored thereoninstructions, which may be used to program a computer system (or otherelectronic devices) to perform a process according to variousembodiments. A non-transitory computer-readable storage medium mayinclude any mechanism for storing information in a form (e.g., software,processing application) readable by a machine (e.g., a computer).Generally speaking, a non-transitory computer-accessible medium mayinclude computer-readable storage media or memory media such as magneticor optical media, e.g., disk or DVD/CD-ROM coupled to computer system2000 via I/O interface 2030. A non-transitory computer-readable storagemedium may also include any volatile or non-volatile media such as RAM(e.g. SDRAM, DDR SDRAM, RDRAM, SRAM, etc.), ROM, etc., that may beincluded in some embodiments of computer system 2000 as system memory2020 or another type of memory. In other embodiments, programinstructions may be communicated using optical, acoustical or other formof propagated signal (e.g., carrier waves, infrared signals, digitalsignals, etc.) conveyed via a communication medium such as a networkand/or a wireless link, such as may be implemented via network interface2040.

In some embodiments, system memory 2020 may include data store 2045,which may be configured as described herein. In general, system memory2020 (e.g., data store 2045 within system memory 2020), persistentstorage 2060, and/or remote storage 2070 may store data blocks, replicasof data blocks, metadata associated with data blocks and/or their state,configuration information, and/or any other information usable inimplementing the methods and techniques described herein.

In one embodiment, I/O interface 2030 may be configured to coordinateI/O traffic between processor 2010, system memory 2020 and anyperipheral devices in the system, including through network interface2040 or other peripheral interfaces. In some embodiments, I/O interface2030 may perform any necessary protocol, timing or other datatransformations to convert data signals from one component (e.g., systemmemory 2020) into a format suitable for use by another component (e.g.,processor 2010). In some embodiments, I/O interface 2030 may includesupport for devices attached through various types of peripheral buses,such as a variant of the Peripheral Component Interconnect (PCI) busstandard or the Universal Serial Bus (USB) standard, for example. Insome embodiments, the function of I/O interface 2030 may be split intotwo or more separate components, such as a north bridge and a southbridge, for example. Also, in some embodiments, some or all of thefunctionality of I/O interface 2030, such as an interface to systemmemory 2020, may be incorporated directly into processor 2010.

Network interface 2040 may be configured to allow data to be exchangedbetween computer system 2000 and other devices attached to a network,such as other computer systems 2090 or various sensors, projectors orother components, for example. In addition, network interface 2040 maybe configured to allow communication between computer system 2000 andvarious I/O devices 2050 and/or remote storage 2070. Input/outputdevices 2050 may, in some embodiments, include one or more displayterminals, keyboards, keypads, touchpads, scanning devices, voice oroptical recognition devices, or any other devices suitable for enteringor retrieving data by one or more computer systems 2000. Multipleinput/output devices 2050 may be present in computer system 2000 or maybe distributed on various nodes of a distributed system that includescomputer system 2000. In some embodiments, similar input/output devicesmay be separate from computer system 2000 and may interact with one ormore nodes of a distributed system that includes computer system 2000through a wired or wireless connection, such as over network interface2040. Network interface 2040 may commonly support one or more wirelessnetworking protocols (e.g., Wi-Fi/IEEE 802.11, or another wirelessnetworking standard). However, in various embodiments, network interface2040 may support communication via any suitable wired or wirelessgeneral data networks, such as other types of Ethernet networks, forexample. Additionally, network interface 2040 may support communicationvia telecommunications/telephony networks such as analog voice networksor digital fiber communications networks, via storage area networks suchas Fibre Channel SANs, or via any other suitable type of network and/orprotocol. In various embodiments, computer system 2000 may include more,fewer, or different components than those illustrated in FIG. 11 (e.g.,displays, video cards, audio cards, peripheral devices, other networkinterfaces such as an ATM interface, an Ethernet interface, a FrameRelay interface, etc.)

Although the embodiments above have been described in considerabledetail, numerous variations and modifications may be made as wouldbecome apparent to those skilled in the art once the above disclosure isfully appreciated. It is intended that the following claims beinterpreted to embrace all such modifications and changes and,accordingly, the above description to be regarded in an illustrativerather than a restrictive sense.

1.-23. (canceled)
 24. A materials handling facility, comprising: one ormore imaging sensors; one or more visual projectors; and a controlsystem, configured to: direct projection of one or more visual task cuesonto the workstation surface as part of a user interface, the visualtask cues associated with performance of an item-handling task;recognize, based at least in part on image data captured via one or moreimaging sensors, an agent gesture at the workstation associated with theperformance of the item-handling task, wherein said recognize includesdetect, based on the image data, an agent gesture with regard to theuser interface projected on the workstation surface as an input actionfor the user interface.
 25. The materials handling facility of claim 24,wherein the control system is further configured to: wherein todetermine the one or more visual task cues for projection, the controlsystem is further configured to: obtain, via one or more imagingsensors, image data at a workstation in a materials handling facility;evaluate the image data of the workstation to recognize one or moreagent gestures at the workstation; and determine the one or more visualtask cues for projection based at least in part on the recognized one ormore agent gestures.
 26. The materials handling facility of claim 24,wherein the control system is further configured to: recognizeadditional agent gestures; and evaluate the recognized additional agentgestures to identify steps in the performance of an assembly taskassociated with the one or more visual task cues.
 27. The materialshandling facility of claim 24, wherein the agent gesture with regard tothe user interface comprises a finger or object placed over a particularinput control of the user interface projected on the workstationsurface.
 28. The materials handling facility of claim 24, wherein thecontrol system is further configured to: obtain, via one or more imagingsensors, additional image data of the workstation surface and geometricinformation of the workstation surface that corresponds to theadditional image data; detect, based at least in part, on the additionalimage data of the workstation surface and the geometric information thatcorresponds to the additional image data obtained from the one or moreimaging sensors, a surface change of the workstation surface thatinterferes with at least one of the projected one or more visual taskcues; determine one or more projection modifications for the at leastone visual task cue; and update the at least one visual task cueprojected onto the workstation according to the one or more projectionmodifications.
 29. A method, comprising: performing, by one or morecomputing devices: obtaining, via one or more imaging sensors, imagedata at a workstation in a materials handling facility; recognizing,based on the image data, an agent gesture at the workstation;determining, based at least in part on the agent gesture, one or morevisual task cues to project onto a workstation surface; and directingthe projection of the one or more visual task cues onto the workstationsurface.
 30. The method recited in claim 29, wherein said recognizingthe agent gesture at the workstation includes detecting the agentgesture with regard to a user interface projected on the workstationsurface as an input action for the user interface.
 31. The methodrecited in claim 30, wherein the input action for the user interface isassociated with the performance of an item-handling task performed atthe workstation.
 32. The method recited in claim 31, wherein theitem-handling task performed at the workstation comprises obtaining ashipping container.
 33. The method recited in claim 29, furthercomprising: determining correctness of motions associated with the taskcues, wherein the task cues include instructions for selecting ashipping container, or instructions for performing one or more tasks ina preferred order or manner.
 34. The method recited in claim 29,obtaining, via the one or more imaging sensors, additional image data ofthe workstation surface and geometric information of the workstationsurface corresponding to the additional image data; based, at least inpart, on the additional image data of the workstation surface and thegeometric information corresponding to the additional image dataobtained from the one or more imaging sensors, detecting a surfacechange of the workstation surface that interferes with at least one ofthe projected one or more visual task cues; determining one or moreprojection modifications for the at least one visual task cue; andupdating the at least one visual task cue projected onto the workstationaccording to the one or more projection modifications.
 35. The methodrecited in claim 29, receiving location information for an agentperforming an item-handling task at the workstation; based, at least inpart, on the location information, determining a viewing perspective forthe agent; and wherein said directing the projection of the one or morevisual task cues onto the workstation surface is performed such that theone or more visual task cues are projected according to the viewingperspective of the agent.
 36. The method recited in claim 29, whereindetermining the one or more visual task cues to project onto theworkstation surface comprises: capturing, by the one or more imagingsensors, the agent gesture with regard to a user interface projected onthe workstation surface as an input action for the user interface; andevaluating the captured agent gesture to identify steps in theperformance of an assembly task being performed at the workstation. 37.A non-transitory, computer-readable storage medium, storing programinstructions that when executed by one or more computing devices causethe one or more computing devices to at least implement: obtaining, viaone or more imaging sensors, image data of a surface of a workstation ina materials handling facility; evaluating the image data of theworkstation surface with regard to a performance of an item-handlingprocess to pack one or more items together into a container at theworkstation based, at least in part on the image data of the workstationsurface; determining, based at least in part on said evaluating theimage data of the workstation surface, a container from a plurality ofdifferent containers; and directing projection of an indication toidentify the selected container for the one or more items onto theworkstation surface.
 38. The non-transitory, computer-readable storagemedium of claim 37, wherein the workstation is a packing workstation,and wherein the item-handling task is a packing process to pack the oneor more items together into a shipping container.
 39. Thenon-transitory, computer-readable storage medium of claim 37, wherein insaid evaluating the image data of the workstation surface with regard tothe performance of the item-handling task at the workstation, theprogram instructions cause the one or more computing devices to at leastimplement: recognizing one or more agent gestures at the workstationassociated with the performance of the item-handling task; based atleast in part on said recognizing the one or more agent gestures at theworkstation surface, determining a correctness of the one or more agentgestures within the context of the item-handling task; and instructingprojection of one or more visual task cues onto the workstation surfacewherein the one or more instructed visual task cues indicate correctnessof the one or more agent gestures for performing the item-handling task.40. The non-transitory, computer-readable storage medium of claim 37,wherein in said evaluating the image data of the workstation surfacewith regard to the performance of the item-handling task at theworkstation, the program instructions cause the one or more computingdevices to at least implement: identifying one or more items located onthe workstation surface associated with the item handling task performedat the workstation, comprising: accessing a data store maintaining itemdescription information regarding items handled in the materialshandling facility; and comparing the image data and the item descriptioninformation to identify the one or more items located on the workstationsurface.
 41. The non-transitory, computer-readable storage medium ofclaim 40, wherein said identifying the one or more items located on theworkstation surface is performed without receiving item information forthe one or more items via a manual item scan.
 42. The non-transitory,computer-readable storage medium of claim 40, wherein the programinstructions further cause the one or more computing devices to at leastimplement: obtaining, via the one or more imaging sensors, geometricinformation of the workstation surface corresponding to the image data;based at least in part on the geometric information and the image data,calculating one or more item dimensions for the one or more items; andwherein said comparing the image data and the item descriptioninformation to identify the one or more items located on the workstationsurface includes comparing the one or more item dimensions as part ofsaid comparing.
 43. The non-transitory, computer-readable storage mediumof claim 40, wherein the program instructions further cause the one ormore computing devices to at least implement: obtaining, via a scaleconfigured to weigh objects on the workstation surface, weightinformation for the workstation surface; wherein said comparing theimage data and the item description information to identify the one ormore items located on the workstation surface includes the weightinformation of the workstation surface as part of said comparison.